Stable Criticality in a Feedforward Neural Network

How do learning processes escape from local optima? Doing so requires an exploration of the landscape at a range of the order of the landscape correlation length – a “long jump” in synapsis space. This brings up a dilemma: because of the high dimensionality of this space, the probability that a random long jump lead to a better optimum is nearly zero. We conjecture that “intelligent” coarse-grained learning operators emerge as a consequence of a self-organization process in neural systems, as follows. The presentation of a single new data vector stimulates the recall of other vectors, each of which generates a small displacement in synapsis space. The sum of these displacements constitutes a coarsegrained learning event. Although long jumps are occasionally needed to escape from local optima, they should be the exception rather than the rule. This leads us to propose a neural network model where the recall process self-organizes to a critical state and one has a power-law distribution in the number of data vectors recalled. * This work is supported in part by CONACyT grant 400349-5-1714E and by the Association Générale pour la Coopération et le Développement (Belgium). 1